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INTRODUCTION 



The Subject 

The controversy surrounding the procurement of the C-5A Galaxy, 
a new transport aircraft obtained for the U.S. Air Force, has become 
a cause celebve for Congressional and Public critics of Department of 
Defense spending. The occurrence of exceptional cost over-runs in 
this program has been cited as evidence of poor management practice. 

One cannot refute this contention, for great excesses in spending 
above budgeted levels per force demonstrate poor management. However, 
it is important to determine which management techniques have failed, 
if circumstances such as those experienced in the Galaxy program are 
to be avoided in the future. 

There is little doubt that the Galaxy program stretched the 
state-of-the-art, to a degree not originally anticipated, in aero- 
nautical engineering and manufacturing, when it sought to reach new 
technological plateaus. It is now evident that economic, as well as 
technological factors, contributed to unforseen developments. The 
initial planning estimates, made in 1964 and based on a preceding 



j 

j 



j 

I 



five-year period marked by a relatively stable economic environment, 
would certainly have been revised upward if conditions experienced 
during the succeeding five years had been anticipated. 

Though the Galaxy program did experience certain technolog- 
ical and economic consequences which might be regarded as extraordinary, 
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it has not been unique in terms of unanticipated expenditures. It 
has been observed that ’’initial cost estimates of major weapons sys- 
tems have historically escalated, just as they have in the C-5A."* 

One should not conclude from the preceding that problems in 
controlling cost increases are peculiar to government. Much of the 
unexpected expense for the Galaxy had to be absorbed by Lockheed Air- 
craft Corporation, the prime contractor, since initial contracts, based 
on Lockheed estimates, did not provide for the expenditures that were 
required. Corporations other than Lockheed have also experienced simi- 
lar predicaments. For example. General Dynamics, while still trying to 
recover from the financial calamities experienced with the F- 111 mili- 
tary aircraft and *the Convair 880-990 commercial jets, is now encoun- 

2 

tering losses in shipbuilding programs; all due to unplanned cost. 

It is possible to gain some insight into the difficulties that 
havfe been described if recognition is given to the commonality which is 
displayed: each of the ventures which has been cited relied heavily on 

research and development to provide techniques and components needed 
for the achievement of program objectives. This fact would then lead 
one to conclude that a major contributor to problems of this type is a 
growing cost of technological advance. 

Improvements in technology derive chiefly from research and 
development efforts. It has been observed, in this regard, that as 
technology does advance, the scientific disciplines, to be considered 
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* Armed Forces Management , (July, 1969), p. 58. 

^"General Dynamics: In Trouble Again," Business Week , (October 

4, 1969), pp. 48-52. 
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in seeking continued advance, become more esoteric in nature and great- 
er in number. Thus, as program objectives move to successively higher 
levels of sophistication, the number of alternatives available for 
realization of the objectives increases, as does the costs of the 
alternatives . 

The problem, then, is one which has increased in significance 
in the face of technological advances, and derives from difficulties 
encountered in both selecting appropriate courses of action for a- 
chieving objectives and in properly forecasting and controlling the 
expense to be realized in pursuing the selected course. It has been 
recognized that there is a growing need for improved techniques to be 
made available to managers for the purpose of making hard choices in 
the funding of research and development efforts. 1 

The Objective 

A major facet of the research and development project man- 
ager's role is essentially one of control. This is exercised by 
monitoring project progress and ensuring that achievement and asso- 
ciated expenditures are kept within planned limits. This function 

will also include the task of affecting certain modifications, as 

2 

necessary, to correct deviations from the plan. 



1 Marvin J. Cetron, "Technological Forecasting: A Prescrip- 

tion for the Military R § D Manager," Naval War College Review , 

XXI (April, 1969), p. 14. 

^William E. Souder, "Experiences with an R 5 D Project Con- 
trol Model," IEEE Transactions on Engineering Management , EM- 15 
(March, 1968), p. 40. 
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Control of research and development is, of course, closely re- 
lated to the planning function which entails the selection of pro- 

I 

grams and the scheduling and allocation of funds. Control and plan- j 

ning interact principly when information, obtained through control, 

! 

causes changes in program scheduling or fund allocation. For instance, 
the progress realized in a program may result in that program being 

i 

terminated entirely, or being expanded at the expense of some other 
program. 



Difficulties are encountered in research and development 
management which are due to an intrinsic uncertainty. Frequently it 
is impossible to discern whether a program is on the verge of failure 
or success. This uncertainty is also quite prevelant in program se- 



lection since a priori estimates are often little more than educated 
guesses, and the selection of programs on the basis of merit is often 
difficult. 

The objective of this paper is to develop a technique for 
evaluating the relative value of proposed programs, so as to assist 
in program selection; for monitoring program progress and evaluating 
observed deviations; and for providing information to be used in ad- 
justment of original planned program content and expenditure levels. 
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Proposed programs are to be assessed on the basis of some selected 

i 

measure of utility so that available funds may be allocated among 

I 

alternative proposals in order to realize a maximum return. Program 

i 

progress is to be measured in terms of achievement versus cost and 
achievement versus time. Thus, it will be possible to identify pro- 
grams that either fail to meet prescribed time schedules or exceed 
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budgeted expenditures. Finally, using data obtained from the analysis 
of program progress, it will be possible to recommend changes in pro- 
gram mix in keeping with observed results. 

As was stated, it is intended that the technique, which is to 
be developed, will be useful only in selecting from among proposed 
programs. It is not intended to formulate proposals. The actual 

i 

formulation of proposals must be done on the basis of objectives and 
applicable policy considerations. This requires a certain amount of 
subjective evaluation. Only after this evaluation has been completed, 
and acceptable candidate programs selected, is it possible to conduct, 
an objective analysis on the basis of the parameters mentioned above. 

In order to facilitate the development of an analytic technique, 
it will be necessary to make certain assumptions regarding the struc- 
ture of subject programs, the statistical independence of programs 
and* the probabilistic character of research and development efforts. 
However, necessary assumptions will be examined to ascertain the 

i 

significance of any inaccuracies which may be introduced. 

The technique which is to be generated will derive from man- 
agement systems common to Department of Defense research laboratories. 

As a result, it will not be universally applicable in all management 

i 



situations. However, since the intent is to simply demonstrate a 
means of incorporating certain analytic methods into the management 
process, the methods themselves should be easily adaptable to other 
management systems. Throughout the ensuing analysis, appropriate 
analogy will be made, as required, to assist in relating the dis- 
cussion to non-defense oriented circumstances. 
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Definitions 

This paper constitutes a merger of several disciplines which 
are of different nature and utilize a variety of nomenclatures. Since 
the discussion is foremost intended as a business-oriented work it is 
assumed that the reader is familiar with the terminology associated 
with management science and economics. Moreover, due to the essential 
quantitative character of the paper, it must be presumed that there is 
also some familiarity with the semantic peculiarities of mathematics; 
particularily probability theory and statistical analysis. However, 
certain terms which lack precise meaning, and are used frequently in 
this paper, do deserve definition as follows: 

1. research and development . — activities which seek to advance 

| 

the state-of-the-art in any technological discipline. The term research 
is usually associated with those activities concerned with basic con- 
cepts and theories and which devote considerable effort to identifying 

I 

and defining topics not previously studied. Development, on the other 

i 

hand, is generally considered to include those activities directed at 

I 

refining and extending knowledge of existing topics. Research and 

i 

i 

development are considered together since research prompts development 

| 

and since both are endeavors which, presumably, have not previously 
been conducted. 

2. project . — any organized research and development activity, 
conducted by a single group and having defined objectives. A project 
is the lowest level of research and development organization, and can- 
not be decomposed into other activities which are capable of satis- 

-£ying_proj e c t_obj ectives . • 
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3. program . --a research and development effort composed of 
several projects. A necessary characteristic of a program is that its 

t 

objectives can be satisfied even if one or more of its constituent 
parts, the projects, are discontinued. 

The Procedure 

Chapter I, "A Review of Current Methods," presents a survey 
of efforts by other authors in developing analytic procedures for use 

i 

in planning and controlling research and development programs. The 
purpose of this survey is to describe the attempts that have been made 

| 

in the past and to evaluate the degree of success that has been real- 

! 

ized for the purpose of identifying difficulties which have been en- 
countered. This survey is also intended to provide a general intro- 

i 

duction to the elements of quantitative management methods. 

Chapter II, "Description of the Technique," examines in depth 
the two principle components of quantitative research and development 
management systems; project selection and control. This description 
is in the form of an examination of various techniques that have been 
developed in the past for use in selection and control. This chapter 
forms the foundation for the techniques to be originated in this paper. 

Chapter III, "Construction of the Model," presents the deriva- 
tion of the selection and control techniques which are the subject of 
the paper. The derivation demonstrates a method of organizing elements 

i 

of the procedures discussed in Chapter II such that an improved ap- 
proach to the problem of project selection is obtained. Also, a j 

technique for optimizing project selection is presented. The chapter 
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concludes with the development of a method for controlling research 
and development projects that is based on statistical inference. 

Chapter IV, "Conclusions,” summarizes the status of quantita- 

! 

tive selection and control techniques as it stands at present, and 
reviews the role that is anticipated for the techniques offered in 
Chapter III. In addition, elements of the subject techniques which 
deserve further refinement are discussed. 

Appended to the text, in Appendix A, is a sample of mat- 



| 

erials which are considered useful in the implementation of quanti- 

I 

tative selection methods. Appendix B presents a tabulation of the 
cumulative Poisson distribution which is used in the control technique 

i 

of Chapter III. Finally, Appendix C presents a partial tabulation of 
the chi-square distribution, also referred to in Chapter III. 



CHAPTER I 



A REVIEW OF CURRENT METHODS 
Focusing On the Problem 

As described in the introduction, the problem under considera- 
tion is one of developing more effective methods for the management of 
research and development programs. A specific provision was included 
which precludes processes involved in the formulation of programs. 

These processes, which are generally classified under the heading of 
normative forecasting, have to do with the identification of goals and 
selection of candidate programs on the basis of policy criteria; usual- 
ly excluding considerations of economy and technological feasibility. 

After the normative forecasting process is complete, and a set 
of acceptable research programs has been generated, management then 
tries to determine which programs from the set to pursue (project se- 
lection), how much money to spend on each program (budget allocation), 
and the size of the total budget for all the programs selected (budget 
determination).* The first two of these functions, project selection 
and budget allocation, constitute the key areas of interest for this 
paper. 

As was stated earlier, project selection is essentially a 

*E. M. Rosen and W. E. Souder, "A Method for Allocating R § D 
Expenditures ," IEEE Transactions On Engineering Management , EM- 12 
(September, 1965), p. 90. 
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planning activity. However, budget allocation is a planning problem 
also; it takes place at the start of a program or at a review point 
where some change of plans is indicated. The key decisions in budget 
(or resource) allocation involve the rates of resource expenditure and 
the planned dates for the commencement of an activity.* 

* It should be re-iterated that, since attention is going to fo- 
cus on planning and plan review, consideration is appropriately due 

i 

ithe associated control processes. In this respect, examination of 

I 

j management processes will include those concerned with monitoring pro- 
gram progress and ensuring that prescribed limits are observed. 



1 



j 

! 
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The Analytic. Framework 

In the abstract, one can consider research as simply the pur- 
chase of information upon which to base later and better decisions. All 
but one of these decisions is whether or not to continue purchasing in- 
formation (that is research). The one exception is the final decision, 
which is whether or not to commercialize the results. The decisions to 
continue a research effort, and the subsequent decisions concerned with 
how to conduct the research do ?ppear to be sequential in nature. This 

sequential decision making process has been the subject of descriptive 
2 

work in the past. It is worthwhile to note that the applicability of 



R. S. Rosenbloom, "Notes on the Development of Network Models 
for Resource Allocation in R § D Projects," IEEE Transactions on Engi- 
neering Management , EM- 11 (June, 1964), p. 62. 

2 

D. L. Marples, "The Decisions of Engineering Design," IRE 
Transactions on Engineering Management, EM-8 (June, 1961), pp. 55-71. 
and T. A. Marshak, "Strategy and Organization in a System Development 
Project," The Rate and Direction of Inventive Activity (Princeton, New 
Jersey: Princeton University Press, 1962), pp. 461-475. 
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the sequential decision model for research efforts is a function of 
the degree of structure that is evidenced in the effort. For example, 
in basic research, which is fairly unstructured, the model should be 
quite appropriate. In developmental design, however, decisions to 
commence a phase of the program may be made before preceding phases 
are complete. In this case, then, some inaccuracies may be intro- 
duced. Nonetheless, for the pruposes of the present discussion, the 
model can be presumed adequate. 

The concept of sequential decisions is only one ingredient in 
a characterization of research and development processes. Previous 

I 

studies, for instance, have identified certain parameters as being 

! 

significant in research efforts. Among these are the magnitude of 

i 

• total expenditures, duration, work scope, activity content, geographic 

| 

dispersion of related elements, and others. Furthermore, any one or 
more of these parameters may be selected as a primary measure; de- 
pending on the type and objectives of the analysis for which the 

i 

characterization is intended. Of the numerous parameters which might 

j 

be used, certain ones have been found to be more useful as constituents 
of a model for the research process. These are: 

1) the items worked on, 

2) the activities (operations) performed on these items, • 

3) the duration of each assignment, 

4) the effort (or resources) utilized by each assignment. 
Furthermore, the dynamics of the research effort have been found to be 
adequately represented by the following: 

1) the start and stop time of each assignment, in calendar 
time, 

2) the total effort content of each assignment. 
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3) the distribution of this effort content over the duration 
of the assignment. 

A final aspect of the research process which should be con- 

i 

sidered in an analytic characterization is the representation of the 
economies of operation. Here, economy of operation relates to the rate 

| 

at which resources can be effectively employed. In this respect, the 
research process is partitioned into three funding stages. The first 

! 

stage lies below a "critical" cost level, such that any allocation be- 
low this level yields no return. Above the critical cost level is a 

i 

stage, within which, progress is a monotonically increasing function 

! ! 

of cost. This mid-stage, which can be called the "adequate" stage, 

! ' i 

, has an upper termination at a funding level called the "satiation" 

i 

i 

cost. Above this cost level increases in cost have no corresponding 

2 | 

increases m progress. 

In concluding this discussion of the analytic approach to re- 
search and development planning and control, it is considered that the 
interactions of the several components of the management system can be 
summarized by graphic display as shown in Figure 1. It is intended 
that this should illustrate the inter- related nature of planning and 

j 

control, and indicate the fashion in which each associates with pro- 
gram operation. 



*P. V. Norden, "On the Anatomy of Development Projects," IRE 
Transaction on Engineering Management , EM-7 (March, 1960), p. 37. 

2 

R. J. Freeman, "A Stochastic Model for Determining the Size 
and Allocation of the Research Budget," IRE Transactions on Engineering 
Management , EM- 7 (March, 1960), p. 4. 
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Some Current Techniques 
Numerous authors have offered quantitative techniques for the 

i 

management of research and development. Of these, the techniques 
that have objectives similar to those of this paper are described below: 

. 1) Asher, 1962 . — This method was derived for the purpose of 
man-power allocation in a pharmaceutical firm. The method considers a 

i 

number of alternative projects on the basis of discounted net value, if 
successful; probability of success; and man-hours (resources) required, 
including skill levels. From these data an expected discounted value 
is computed. Next, linear programming is used to assign various re- 
search teams, characterized by skill -levels represented, to maximize 

the expected return subject to a constraining number of hours available 
2 I 

for each team. 

2) Atkinson and Bobis, 1969 . --This is a method for determin- 
ing money to be spent on product oriented research programs. The 
method commences by estimating the density function of the probability 

! 

of a project succeeding for a given expenditure. This distribution 

i 

is given in the form of a logistic function. On the basis of this 
function, the probability of completing a program within a given time, 
for a given expenditure, is determined. 



*For a brief but thorough review of current techniques, in- 
cluding an exceptional bibliography of the topic see M. J. Cetron, J. 
Martino and L, Roepcke, "The Selection of R 6 D Program Content — Survey 
6f Quantitative Methods,” IEEE Transactions on Engineering Management , 
EM-14 (March, 1967), pp. 4-13. 

2 

D. T. Asher, ”A Linear Programming Model for the Allocation of 
R and D Efforts,” IRE Transactions on Engineering Management , EM-9 
(December, 1962), pp. 154-57. j 
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The next step in the analysis involves a determination of com- | 

i 

mercial value as a function of completion time. This, in conjunction 
with estimates of completion time yields an expected commercial value. 
Having these data, it is then possible to optimize the expected value 

| 

of research on the basis of annual expenditures for the alternative 
programs being considered. Due to the formidable nature of this op- 
timization problem which results from the numerous variables involved, 

' i i 

an iterative optimization technique is employed. 

i I 

3) Dean and Hauser, 1967 . --The employment of this method com- 
mences with a decomposition of the research program into the following 
catagories : 

a) Qualitative Materiel Development Objectives (QMDO)--the 

; i 

several major goals of the research (effort); corresponding approximate- 

! | 

ly to program objectives as defined in the Introduction. 

\ ! 

b) Materiel Concepts--the various different component efforts 

■ 

of the QMDO (for instance, if a QMDO is the development of a missile, 

j I 

I these might be guidance, propulsion and structure); corresponding 
approximately to program content. 

c) Technical Approaches — the several alternative methods of 
achieving a materiel concept; corresponding to different projects 
which satisfy the same objectives. 

The utility analysis of the research program is affected by 
first estimating the probability of success and cost for the technical 

*A. C. Atkinson and A. H. Bobis, "A Mathematical Basis for the 
Selection of Research Projects,” IEEE Transactions on Engineering 
Management , EM-16 (February, 1969), pp. 2-8. 
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approaches. With these data it is then possible to determine the al- 
location of funds, among the technical approaches for a given material 
concept, which maximizes the probability of the material concept suc- 
ceeding; subject to the constraint of a fixed amount being available 
for the material concept funding. 

Assuming, that, in order for a QMDO to succeed, all component 

materiel concepts must succeed, it is then possible to find the fund- 

j 

ing levels for materiel concepts in order to maximize the QMDO proba- 
bility of success; subject to a budget constraint. 

This particular method has been found amenable to optimization 



by dynamic programming. The authors maintain that such a solution 

I 

contributes flexibility and speed to solution, and is particularly 

i 

suitable for the type of structure used in decomposition of the 

i 

program. 1 

4) Hess, 1962 . --Hess introduced the first dynamic programming 



| solution to the allocation problem. His solution assumes values for 
certain parameters, among them being the probability of technical 
success, expected profit as a function of time of completion, and an 
assumed measure of usefulness deriving from prior research. Using the 
dynamic technique that is developed, it is possible to obtain an op- 

j 

timal allocation of funds among programs and budgeted rates of ex- 



penditure for each program; subject to appropriate overall budget 



1 B. V. Dean and L. E. Hauser, "Advanced Materiel Systems Plan- 
ning," IEEE Transactions on Engineering Management, EM-14 (March, 1967), 
pp. 21-43. 



17 



constraints . * 



Aspects of Uncertainty 

Complications attendent to uncertainty arising from opera- 
tional situations are by no means limited to management of research 

i 

and development activities. Similar problems are common to the 
management of most business efforts. This is evidenced by the de- 
velopment of disciplines such as statistical decision theory, and 
their rise to prominence in management science. Research efforts, 
however, are somewhat unique in that they demonstrate a degree of 

i 

endogenous uncertainty not characteristic of other management pro- 
cesses. Even if environmental factors (such as market conditions and 
the supply of materials, labor and funds) were deterministic, the re- 
sults of research and development programs would not be. Thus, the 
significance of uncertainty in research management has added impor- 
tance. 

As is indicated by the preceding description of some available 
techniques, this uncertainty is usually accounted for by means of a 

. 

"probability of success" term. This approach to the problem, however, 

i 

has some disadvantages. J 

To begin with, there is no universally accepted meaning of 
probability of success. Most authors account for this, to some extent, 

| 

by allowing for a probability of technical success and, also, for 

Is. W. Hess, "A Dynamic Programming Approach to R and D Budg- 
eting and Project Selection," IRE Transactions on Engineering Manage- 
ment , EM-9 (December, 1962), pp. 170-79. 
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commercial success. Even if one were to limit consideration to 
technical success alone, a probability measure is inadequate, since it 
does not allow for varying degrees of success which might result. In 
reference to the probability of success concept, Baker and Pound ob- 
served that "at best, it is extremely difficult to understand what is 
meant by such a term."l 

The use of probability of success is further complicated by 
the fact that it is extremely difficult to estimate a value for such a 
parameter. In the work done by Asher, for instance, probabilities 

i 

j ranged between five and twenty-five chances of success per ten 
thousand . 2 Needless to say, even the' slightest variation in estimates, 
in cases such as this, would have disproportionate effects due to the 



i 



i 



i 

small order of magnitude involved. There are two principle methods 

of estimating probability of success. First is by obtaining the sub- 

I 

jective judgements of persons experienced in the subject area. Second 
is by analysis of data obtained from similar prior efforts. The main 
shortcoming of the first method lies in the bias that is likely to be 

I 

introduced. The second is found disadvantageous since probability of 

i 

success is not a fixed quantity, but is a function of the state-of- 

! 

the-art and level of knowledge prevailing at the time of the estimate. ^ 



*N. R. Baker and W. H. Pound, "R § D Project Selection: Where 

We Stand," IEEE Transactions on Engineering Management, EM-11 (December, 
1964), p. 129. 

^ Asher, "A Linear Programming Model for the Allocation of R § D 



Efforts," p. 155. 




^Rosen and Souder, "A Method for Allocating R 5 D Expenditures," 



p. 87. 
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. A final complication realized in using probability of success 
estimates is that there is no reason to assume that the probability of 
success and level of funding are independent. On the contrary, the 
converse is almost surely true. Despite this, only the method of Hess 
gives consideration to the apparent dependency. 

Data Requirements 

Since the descriptions of methods given earlier were somewhat 
abbreviated, the data requirements appear deceptively minor. In gen- 
eral all techniques, developed to date, require information of past 
.efforts. As such, many of the methods are not applicable to organiza- 
tions that are initiating research and development programs. In some 
teases, such as that of Hess, the data requirements and assumptions are 
quite restrictive. As Hess points out, a considerable amount of work 
would be required before his method could be applied. 1 

The fact of the matter is that, at the present time, there is 
not sufficient data available upon which to base formally structured 
research and development selection decisions. Even if a great deal of 
past data were available, it is not clear how this might be useful in 
evaluating current, different projects; for reasons which have already 
been discussed. 

As a result of this paucity of "good" information, compensatory 
methods have been undertaken. As was mentioned in the discussion of 
probability, one approach has been the use of subj ective .estimates , an 

* Baker and Pound, "R § D Project Selection: Where We Stand,’" 

p. 128. 



